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Abstract.  Traditional  biometric  recognition  systems  rely  on  a  single 
biometric  signature  for  authentication.  While  the  advantage  of  using  mul¬ 
tiple  sources  of  information  for  establishing  the  identity  has  been  widely 
recognized,  computational  models  for  multimodal  biometrics  recognition 
have  only  recently  received  attention.  We  propose  a  novel  multimodal 
multivariate  sparse  representation  method  for  multimodal  biometrics 
recognition,  which  represents  the  test  data  by  a  sparse  linear  combina¬ 
tion  of  training  data,  while  constraining  the  observations  from  different 
modalities  of  the  test  subject  to  share  their  sparse  representations.  Thus, 
we  simultaneously  take  into  account  correlations  as  well  as  coupling  in¬ 
formation  between  biometric  modalities.  Furthermore,  the  model  is  mod¬ 
ified  to  make  it  robust  to  noise  and  occlusion.  The  resulting  optimization 
problem  is  solved  using  an  efficient  alternative  direction  method.  Exper¬ 
iments  on  a  challenging  public  dataset  show  that  our  method  compares 
favorably  with  competing  fusion-based  methods. 


1  Introduction 

Unimodal  biometric  systems  rely  on  the  evidence  of  a  single  source  of  information 
such  as  a  single  iris  or  fingerprint  or  face  for  authentication.  Unfortunately  these 
systems  often  have  to  deal  with  some  of  the  following  inevitable  problems  [1]:  (a) 
Noisy  data  (b)  Non-universality:  the  biometric  system  based  on  a  single  source 
of  evidence  may  not  be  able  to  capture  meaningful  data  from  some  users,  (c) 
Intra-class  variations:  in  the  case  of  iris  recognition  a  user  who  wears  artificial 
contact  lenses  with  various  patterns  can  cause  these  variations,  (d)  Spoof  attack: 
hand  signature  forgery  is  an  example  of  this  type  of  attack.  It  has  been  observed 
that  some  of  the  limitations  of  unimodal  biometric  systems  can  be  addressed  by 
deploying  multimodal  biometric  systems  that  essentially  integrate  the  evidence 
presented  by  multiple  sources  of  information  such  as  iris,  fingerprints  and  face. 

Classification  in  multibiometric  systems  is  done  by  fusing  information  from 
different  modalities.  The  information  fusion  can  be  done  at  different  levels,  which 
can  be  broadly  divided  into  feature  level,  score  level  and  rank/decision  level  fu¬ 
sion.  Due  to  preservation  of  raw  information,  feature  level  fusion  can  be  more 
discriminative  than  score  or  decision  level  fusion  Q.  But,  there  have  been  very 
little  effort  in  exploring  feature  level  fusion  in  the  biometric  community.  This  is 
because  of  the  different  output  formats  of  different  sensors,  which  result  in  fea¬ 
tures  with  different  dimensions.  Often  the  features  have  large  dimensions,  and 
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fusion  becomes  difficult  at  feature  level.  The  prevalent  method  is  feature  concate¬ 
nation,  which  has  been  used  for  different  multibiometric  settings  ill  .  However, 
in  many  scenarios,  each  modality  produces  high-dimensional  features.  In  such 
cases,  the  method  is  both  impractical  and  non-robust.  It  also  cannot  exploit  the 
constraint  that  features  of  different  modalities  should  share  the  identities. 

In  recent  years,  theories  of  Sparse  Representation  (SR)  and  Compressed  Sens¬ 
ing  (CS)  have  emerged  as  powerful  tools  for  efficiently  processing  data.  This 
has  led  to  a  resurgence  in  interest  in  the  principles  of  SR  and  CS  for  biomet¬ 
rics  recognition.  See  Hi  and  the  references  therein  for  a  survey  of  biometrics 
recognition  algorithms  using  SR  and  CS.  Motivated  by  the  success  of  SR  in 
unimodal  biometric  recognition,  we  propose  a  joint  sparsity-based  algorithm  for 
multimodal  biometrics  recognition.  Our  method  is  based  on  the  well  known  regu¬ 
larized  regression  method,  multi-task  multivariate  Lasso  @,6]-  Figure. [l] presents 
an  overview  of  our  method. 

This  paper  makes  the  following  contributions: 

—  We  present  a  robust  feature  level  fusion  algorithm  for  multibiometric  recog¬ 
nition  tasks.  Through  the  proposed  joint  sparse  framework,  we  can  easily 
handle  different  dimensions  of  different  modalities  by  forcing  the  different 
features  to  interact  through  their  sparse  coefficients.  Further,  the  proposed 
algorithm  can  efficiently  handle  large  dimensional  feature  vectors. 

—  We  make  the  classification  robust  to  occlusion  and  noise  by  introducing  an 
error  term  into  the  optimization  framework. 

—  The  algorithm  is  easily  generalizable  to  handle  multiple  test  inputs  from  a 
modality. 


Fig.  1.  Overview  of  our  algorithm 
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2  Joint  Sparsity-Based  Multimodal  Biometrics 
Recognition 


Consider  a  multimodal  C-class  classification  problem  with  D  different  biometric 
traits.  Suppose  there  are  pi  training  samples  in  each  biometric  trait.  For  each 
biometric  trait  i  =  1, . . . ,  U,  we  denote 

Xi  [yi  y i  y i  l 

—  •  •  •  7  ^C\ 

as  an  n  x  pi  dictionary  of  training  samples  consisting  of  C  sub-dictionaries  X^’s 
corresponding  to  C  different  classes.  Each  sub-dictionary 


X\  —  Iy2  y2. 

3  5 1  ^  ^~j  •)  2  5 


.w 


e  Rnxpj 


represents  a  set  of  training  data  from  the  ith  modality  labeled  with  the  j th  class. 
Note  that  n  is  the  feature  dimension  of  each  sample  and  there  are  pj  number  of 
training  samples  in  class  j.  Hence,  there  are  a  total  of  p  =  JN=1  pj  many  samples 
in  the  dictionary  X^.  In  multimodal  biometrics  recognition  problem  given  a  test 
samples  (matrix)  Y,  which  consists  of  D  different  modalities  {Y1,  Y2, . . . ,  YD} 
where  each  sample  Y2  consists  of  d{  observations  Y2  =  [yi,  y2, . . . ,  y^]  E  Rnxdi, 
the  objective  is  to  identify  the  class  to  which  a  test  sample  Y  belongs  to.  In 
what  follows,  we  present  a  multimodal  multivariate  sparse  representation-based 
algorithm  for  this  problem  @0. 


2.1  Multimodal  Multivariate  Sparse  Representation 

We  want  to  exploit  the  joint  sparsity  of  coefficients  from  different  biometrics 
modalities  to  make  a  joint  decision.  To  simplify  this  model,  let  us  consider  a 
bimodal  classification  problem  where  the  test  sample  Y  =  [Y1,  Y2]  consists  of 
two  different  modalities  such  as  iris  and  face.  Suppose  that  Y1  belongs  to  the  j  th 
class.  Then,  it  can  be  reconstructed  by  a  linear  combination  of  the  atoms  in  the 
sub-dictionary  Xj.  That  is,  Y1  =  X1/''1  +N1,  where  T1  is  a  sparse  matrix  with 
only  pj  nonzero  rows  associated  with  the  j  th  class  and  N1  is  the  noise  matrix. 
Similarly,  since  Y2  represents  the  same  subject,  it  belongs  to  the  same  class  and 
can  be  represented  by  training  samples  in  X2  with  different  set  of  coefficients 
r2j.  Thus,  we  can  write  Y2  =  X2T2  +  N2,  where  r2  is  a  sparse  matrix  that 
has  the  same  sparsity  pattern  as  T1.  If  we  let  T  =  [T1,  T2],  then  T  is  a  sparse 
matrix  with  only  pj  nonzeros  rows. 

In  the  more  general  case  where  we  have  D  modalities,  if  we  denote  {Y2}^ 
as  a  set  of  D  observations  each  consisting  of  di  samples  from  each  modality  and 
let  r  =  [T1,  T2, . . . ,  rD]  G  Rpxd  be  the  matrix  formed  by  concatenating  the 
coefficient  matrices  with  d  =  di,  then  we  can  seek  the  row-sparse  matrix 

r  by  solving  the  following  ^/^-regularized  least  square  problem 

t  =  argmin  1  £  ||V  -  X>r\\2F  +  A||r||lig 

i=  1 


(i) 
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where  A  is  a  positive  parameter  and  q  is  set  greater  than  1  to  make  the  optimiza¬ 
tion  problem  convex.  Here,  ||T||i5g  is  a  norm  defined  as  ||-T|| —  YX=i  II lk\\q 
where  7^’s  are  the  row  vectors  of  T  and  ||Y||i?  is  the  Frobenius  norm  of  matrix 

Y  defined  as  ||Y||F  =  y Y?y  Once  t  is  obtained,  the  class  label  associated 

with  an  observed  vector  is  then  declared  as  the  one  that  produces  the  smallest 
approximation  error 


D 

3  =  argmin]T  ||Y*  -  Xi«5}(ri)fF,  (2) 

3  i=  1 

where  Sj  is  the  matrix  indicator  function  defined  by  keeping  rows  corresponding 
to  the  j th  class  and  setting  all  other  rows  equal  to  zero.  Note  that  the  optimiza¬ 
tion  problem  ©  reduces  to  the  conventional  Lasso  [10]  when  D  =  1  and  dm  1. 
For  Dm  1  (jTJ) ,  it  is  equivalent  to  multivariate  Lasso  [7|. 


2.2  Robust  Multimodal  Multivariate  Sparse  Representation 

In  this  section,  we  consider  a  more  general  problem  where  the  data  is  contami¬ 
nated  by  noise.  In  this  case,  the  observation  model  can  be  modeled  as 

Yi  =  XiTi  +  Z*  +  N\  i  =  l,...D,  (3) 

where  N2,  is  a  small  dense  additive  noise  and  Z2,  E  Wixdi  is  a  matrix  of  back¬ 
ground  noise  (occlusion)  with  arbitrarily  large  magnitude.  One  can  assume  that 
each  Z2,  is  sparsely  represented  in  some  basis  EF  E  Mnxm\  That  is,  Z2,  =  B lAl 
for  some  sparse  matrices  A1  E  Hence,  ©  can  be  rewritten  as 

Y^XLP  +  BLA^  +  N*,  i  =  (4) 

With  this  model,  one  can  simultaneously  recover  the  coefficients  rl  and  A 1  by 
taking  advantage  of  that  fact  that  A 1  are  sparse 

i  D 

t.  A  =  argmin  -  ||Y*  -  -  B lAl\\2F  + 

’  Z  i= 1 

Ai||r||i?,  +  A2||A||i,  (5) 

where  Ai  and  A2  are  positive  parameters  and  A  =  [A1,  A2, . . . ,  AD ]  is  the  sparse 
coefficient  matrix  corresponding  to  occlusion.  The  ^i-norm  of  matrix  A  is  defined 
as  || A ||i  =  J2ij  \Ai,j\-  Note  that  the  idea  of  exploiting  the  sparsity  of  occlusion 
term  has  been  studied  by  Wright  et  al.  Q. 

Once  T1,  A  are  computed,  the  effect  of  occlusion  can  be  removed  by  setting 
Yl  =  Yl  —  B 1  A1 .  One  can  then  declare  the  class  label  associated  to  an  observed 
vector  as 

D 

j  =  argmin^  ||Y*  -  -  WA^. 


(6) 
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2.3  Optimization  Algorithm 

In  this  section,  we  present  an  algorithm  to  solve  (0  based  on  the  classical  alter¬ 
nating  direction  method  of  multipliers  (ADMM)  |llj|.  0.  Note  that  the  opti¬ 
mization  problem  ©  can  be  solved  by  setting  A2  equal  to  zero.  Let 


i=  1 


In  ADMM  the  idea  is  to  decouple  C(r,  A),\\r\\iiq  and  ||A||i  by  introducing 
auxiliary  variables  to  reformulate  the  problem  into  a  constrained  optimization 
problem 


r mmvC(r,  A)  +  Ai||V||i,g  +  A2||U||i  s.  t. 

r  =  v,  a  =  u. 


(7) 


Since,  (0  is  an  equally  constrained  problem,  the  Augmented  Lagrangian  method 
(ALM)pjJ  can  be  used  to  solve  the  problem.  This  can  be  done  by  minimizing 
the  augmented  lagrangian  function  /ar,aA(T,  A ,  V,  U;  A^,  Ap)  defined  as 


C(r,  A)  +  A2 1| U ||i  +  {A a,  A  —  U)  +  -±\\A  —  V\\* 1 2 3 4 5 6F+ 
Ai||V||m  +  (Ar,  r-V)  +  ^\\r-  v\\%, 


(8) 


where  A  a  and  Ap  are  the  multipliers  of  the  two  linear  constraints,  and  O/p  op  are 
the  positive  penalty  parameters.  The  ALM  algorithm  solves  /ar,aA(T,  A,  V,  U; 
A  a,  Ap)  with  respect  to  F,  A ,  U  and  V  jointly,  keeping  Ap  and  A  a  fixed  and  then 
updating  Ap  and  A  a  keeping  the  remaining  variables  fixed.  Due  to  the  separable 
structure  of  the  objective  function  far,aA  >  one  can  further  simplify  the  problem  by 
minimizing  far, a  A  with  respect  to  variables  T,  A,  U  and  V,  separately.  Different 
steps  of  the  algorithm  are  given  in  Algorithm[l]  In  what  follows,  we  describe  each 
of  the  suboptimization  problems  in  detail. 


Algorithm  1:  Alternating  Direction  Method  of  Multipliers  (ADMM). 

Initialize:  F 0,  U0,  V0,  Aa,o,  Ar,o,  ar,aA 
While  not  converged  do 

1.  Ft+i  =  argminr /ar,aA(T,  At,Ut,  Vt;  Ar,t,  Aa,t) 

2.  At- 1-1  =  aigminA  far, aA(rt+i,  A,  Ut,  Vt;  Apt,  AA,t) 

3.  Ut+i  =  argminu  far,aA  (A+i,  At+i,  U,  Vt;  Apt,  AA,t) 

4.  Vt+i  =  arg minv  far,aA(Ft+i,  At+i,  Ut+i,  V;  Apt,  AA,t) 

5.  Ar,t+i  ==  Apt  +  OLA(rt+ 1  —  Ut+i) 

6.  Aa,£+i  =  A A,t  +  car  (Ftp  1  —  Vt+i) 
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Update  Step  for  T:  The  first  suboptimization  problem  involves  the  mini¬ 
mization  of  far, a  A  AV,U;  A/p  Af)  with  respect  to  T.  It  has  the  quadratic 
structure,  which  is  easy  to  solve  by  setting  the  first-order  derivative  equal  to 
zero,  and  has  the  following  solution 

r\+1  =  (XiTX‘  +  arl)-\-lVT {Yl  -  Ai)  +  arY\  +  A^t), 

where  I  is  p  x  p  identity  matrix  and  AJ,  VJ  and  Ay  t  are  submatrices  of  At,\t 
and  Ay?t5  respectively. 

Update  Step  for  A:  The  second  suboptimization  problem  is  similar  in  nature, 
whose  solution  is  given  below 

Aj+1  =  (1  +  aA)~1(Yi  -  +  aAUl  -  A^), 

where  UJ  and  A^  t  are  submatrices  of  Ut  and  A A,t,  respectively. 

Update  step  for  U  :  The  third  suboptimization  problem  is  with  respect  to  U, 
which  is  the  standard  £i  minimization  problem  which  can  be  recast  as 

mm  h\At+1  +  a^AAA  -  U|||  +  y-||U||i.  (9) 

U  Z  (A  A 

Equation  ([9|)  is  the  well-known  shrinkage  problem  whose  solution  is  given  by 

Um  =  S  (^At+l  +  &A  ^  5 

where  <S(a,  b )  =  sgn(a)(\a\  —  b)  for  \a\  >  b  and  zero  otherwise. 

Update  Step  for  V  :  The  final  suboptimization  problem  is  with  respect  to  V 
and  can  be  formulated  as 

min  hrt+1  +  apAr,t  -  V||*  +  ^-||V||li9.  (10) 

V  Z  OLp 

Due  to  the  separable  structure  of  <nnj>,  it  can  be  solved  by  minimizing  with 
respect  to  each  row  of  V  separately.  Let  7it+1,  ar,i,t  and  v^t+i  be  rows  of 
matrices  Tt+i,  Ar,t  and  Vt+ 1,  respectively.  Then  for  each  i  =  1, . . .  ,p  we  solve 
the  following  sub-problem 

Vi,t+ 1  =  argmin  I||z  -  v|||  +  ??||v||g,  (11) 

v  Z 

where  z  =  7 i  t+1  —  a r^t&r1  and  77  =  One  can  derive  the  solution  for  (pTTj) 
for  any  q.  In  this  paper,  we  only  focus  on  the  case  when  q  =  2.  The  solution  of 
ciD  has  the  following  form 


where  (v)+  is  a  vector  with  entries  receiving  values  max(^,0). 
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Our  algorithm  for  multimodal  biometrics  recognition  is  summarized  in  Algo¬ 
rithm  [2j 


Algorithm  2:  Sparse  Multimodal  Biometrics  Recognition  (SMBR). 

Input:  Training  samples  {Xi}^L1?  test  sample  {Yi}^,  Occlusion  basis  {B}^ 
Procedure:  Obtain  P  and  A  by  solving 

i  D 

r,A  =  argmin  -  y]  ||V  -  XT*  -  B*jT|||.  +  Ai||r||lj?  +  A2||yl||i, 

’  i=  1 

Output:  identity (Y)  =  argmuq  ||Yl  —  ^C8){pZ)  —  BMZ||^. 


3  Experiments 

We  evaluated  our  algorithm  for  different  multi-biometric  settings.  We  tested  on 
a  publicly  available  dataset  -  the  WVU  Multimodal  dataset  [13|.  The  WVU 
dataset  is  one  of  the  few  publicly  available  datasets  which  allows  fusion  at  image 
level,  hence  the  proposed  feature  level  fusion  technique  can  be  tested.  In  all  the 
experiments,  B^  was  set  to  be  identity  for  convenience,  z.e.,  we  consider  error  to 
be  sparse  in  image  domain. 

3.1  WVU  Multimodal  Dataset 

WVU  multimodal  dataset  is  a  comprehensive  collection  of  different  biometric 
modalities  such  as  fingerprint,  iris,  palmprint,  hand  geometry  and  voice  from 
subjects  of  different  age,  gender  and  ethnicity.  It  is  a  challenging  dataset  and 
many  of  these  samples  are  of  poor  quality  corrupted  with  blur,  occlusion  and 
sensor  noise  as  shown  in  Figure  [2j  Out  of  these,  we  chose  iris  and  fingerprint 
modalities  for  testing  the  algorithm,  giving  a  total  of  6  modalities  (4  fingerprint 
+  2  iris).  The  evaluation  was  done  on  a  common  subset  of  219  subjects  having 
samples  in  all  the  modalities. 


Fig.  2.  Examples  of  challenging  images  from  the  WVU  Multimodal  dataset,  suffering 
from  various  artifacts  as  sensor  noise,  blur,  occlusion  and  poor  acquisition 

Preprocessing.  Robust  pre-processing  of  images  was  done  before  feature  ex¬ 
traction.  Iris  images  were  segmented  following  the  recent  method  proposed  in 
0.  Following  the  segmentation,  25  x  240  iris  templates  were  formed  by  re¬ 
sampling  using  the  publicly  available  code  of  Masek  et  al.  [15].  Fingerprint 
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images  were  enhanced  using  the  filtering  based  methods  described  in  0  ,  and 
then  the  core  point  was  detected  using  the  enhanced  images  0  .  Features  were 
then  extracted  around  the  detected  core  point. 

Feature  Extraction.  Gabor  features  were  extracted  on  the  processed  images 
as  they  have  been  shown  to  give  good  performance  on  both  fingerprints  0  and 
iris  |15[.  For  fingerprints,  the  processed  images  were  convolved  with  Gabor  filters 
at  8  different  orientations.  Circular  tesselations  were  extracted  around  the  core 
point  for  all  filtered  images.  The  mean  values  for  each  sector  were  concatenated 
to  form  3600  x  1  feature  vectors.  For  iris,  the  templates  were  convovled  with  a 
log-Gabor  filter,  and  vectorized  to  give  6000  x  1  dimensional  feature. 


Experimental  Set-up.  The  dataset  was  randomly  divided  into  4  training  sam¬ 
ples  per  class  (1  sample  here  is  1  data  sample  each  from  6  modalities)  and  the 
rest  519  for  testing.  The  recognition  result  was  averaged  over  5  runs.  The  pro¬ 
posed  methods  were  compared  with  compared  with  state-of-the-art  classification 
methods  such  as  sparse  logistic  regression  (SLR)  [lgf  and  SVM  0.  Although 
these  methods  give  superior  performance  on  individual  modalities,  they  cannot 
handle  multimodal  data.  One  possible  way  to  handle  multimodal  data  is  to  use 
feature  concatenation.  But,  this  resulted  in  feature  vectors  of  size  26400  x  1 
when  all  6  modalities  are  used,  and  is  not  useful.  Hence,  two  techniques  were  ex¬ 
plored  for  fusion.  In  the  first  technique,  a  score-based  fusion  was  followed  where 
the  probability  outputs  for  test  sample  of  each  modality,  {yi}f=1  were  added  to¬ 
gether  to  give  the  final  score  vector.  Classification  was  based  upon  the  final  score 
values.  For  the  second  technique,  the  subject  chosen  by  the  maximum  number 
of  modalities  was  taken  to  be  from  the  correct  class. 


Observations.  The  recognition  performances  of  SMBR-WE  (without  error) 
and  SMBR-E  (with  error)  were  compared  with  linear  SVM  and  linear  SLR  clas¬ 
sification  methods.  In  the  experiments,  Ai  and  A2  were  set  to  0.01.  Figures  [3] 
and  Table  [l]  demonstrate  recognition  performance  of  different  methods  on  three 
fusion  settings  -  (1)  two  irises,  (2)  four  fingerprints  and  (3)  all  combined. 


Table  1.  Rank  one  recognition  performance  for  WVU  Multimodal  dataset 


SMBR-WE 

SMBR-E 

SLR-Sum 

SLR- Major 

SVM-Sum 

SVM-Major 

4  Fingerprints 

97.9 

97.6 

96.3 

74.2 

90.0 

73.0 

2  Irises 

76.5 

78.2 

72.7 

64.2 

62.8 

49.3 

Overall 

98.7 

98.6 

97.6 

84.4 

94.9 

81.3 

Comparsion  of  Methods:  Clearly,  the  proposed  SMBR  approach  outperforms  ex¬ 
isting  techniques  for  all  the  fusion  settings.  Both  SMBR-E  and  SMBR-WE  have 
similar  performance,  though  the  latter  seems  to  give  a  slightly  better  perfor¬ 
mance.  This  may  be  due  to  the  penalty  on  the  sparse  error,  though  the  error 
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Fig.  3.  CMC  (Cumulative  Match  Curve)  for  multimodal  fusion  using  (a)  four  finger¬ 
prints,  (b)  two  irises  and  (c)  all  modalities 


may  not  be  sparse  in  image  domain.  Further,  sum-based  fusion  shows  a  superior 
performance  over  voting-based  methods,  and  SLR  performs  better  than  SVM 
on  all  the  modalities.  However,  by  jointly  classifying  all  the  modalities,  SMBR 
achieves  the  best  performance. 

4  Conclusion 

We  have  proposed  a  novel  multimodel  multivariate  joint  sparsity-based  algo¬ 
rithm  for  multimodal  biometrics  recognition.  The  algorithm  is  robust  as  it  ex¬ 
plicitly  accounts  for  both  noise  and  occlusion.  An  efficient  algorithm  based  on 
alternative  direction  was  proposed  for  solving  the  optimization  problem.  Vari¬ 
ous  experiments  show  that  our  method  is  robust  and  significantly  improves  the 
overall  recognition  accuracy. 
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